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Preface 


RAND 


A  growing  number  of  states  are  implementing  standards-based 
accountability  systems  in  efforts  to  improve  student  achievement. 
Policymakers  in  these  states  believe  that  standards-based  reforms 
that  include  high-stakes  testing  can  be  powerful  tools  to  change 
what  is  happening  in  schools  and  classrooms.  This  study  is  part 
of  ongoing  research  funded  by  the  National  Center  for  Research 
on  Evaluation,  Standards,  and  Student  Testing  (CRESST)  at  the 
University  of  California,  Los  Angeles  (UCLA)  to  understand  the 
impact  of  standards-based  state  education  reforms  on  schools  and 
classrooms. 

Researchers  from  RAND  and  the  University  of  Colorado  (CU)- 
Boulder  began  this  program  of  research  in  1995-96  with  an 
examination  of  the  effects  of  standards-based  education  reform  in 
Kentucky.  Kentucky's  reform,  which  was  one  of  the  earliest  in 
the  country,  included  new  performance  standards  for  students 
and  a  new  statewide  testing  system.  The  testing  system,  called  the 
Kentucky  Instructional  Results  Information  System  (KIRIS),  was 
exclusively  performance-based,  including  mathematics  and 
writing  portfolios  as  well  as  other  open-response  tasks.  The 
education  reform  also  had  a  strong  accountability  component; 
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schools  showing  adequate  improvement  were  given  cash  awards,  and 
schools  whose  scores  declined  were  declared  "in  crisis,"  provided  with 
assistance,  and  if  they  continued  in  crisis  for  an  extended  period, 
threatened  with  reorganization. 

After  studying  the  effects  of  the  Kentucky  reform  for  three  years,  the 
researchers  shifted  their  attention  to  Washington,  a  state  that  was  just 
beginning  to  implement  standards-based  reform.  Washington 
provided  a  number  of  interesting  contrasts  with  Kentucky.  For 
example,  Washington  chose  to  implement  changes  gradually  over  a 
period  of  a  decade  rather  than  all  at  once.  Also,  because  of  a  strong 
tradition  of  local  control,  the  Washington  reform  left  more 
responsibility  for  implementation,  particularly  for  professional 
development,  in  the  hands  of  local  districts.  Research  activities  began 
in  Washington  in  1998-1999  and  will  continue  through  1999-2000. 

This  briefing  summarizes  results  from  the  1999  survey  of  teachers  and 
students.  The  full  report  is  available  from  the  National  Center  for 
Research  on  Evaluation,  Standards,  and  Student  Testing  (CRESST)  at 
UCLA:  http://www.cse.ucla.edu/-1 

An  electronic  version  of  the  slides  in  this  briefing  is  available  on  the 
web  site  of  Washington  state's  Office  of  the  Superintendent  of  Public 
Instruction:  http://www.kl2.wa.us/puborder/order.asp. 


1B.M.  Stecher,  S.L.  Barron,  T.  Chun,  and  K.  Ross  (in  press).  The  Effects  of 
Washington  State  Education  Reform  on  Schools  and  Classrooms.  CSE  Technical 
Report  525.  Los  Angeles,  CA:  UCLA  Center  for  the  Study  of  Evaluation. 
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Washington  principals  and  teachers,  who  contributed  their  time  to 
complete  our  surveys.  We  also  want  to  express  our  gratitude  to  Dr. 
Terry  Bergeson,  Washington  Superintendent  of  Public  Instruction,  and 
to  her  staff  for  their  support  and  assistance.  In  particular,  Shirley 
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J.  D.  Lieber,  Raylette  Pickett,  and  Laura  Quintero.  Donna  White 
helped  in  the  preparation  of  this  document.  Gail  Zellman  reviewed 
the  document. 
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Researchers  from  RAND  are  studying  the  implementation  and  impact 
of  the  Washington  reform  on  school  and  classroom  practices,  focusing 
on  the  subjects  of  writing  and  mathematics.  This  briefing  presents 
findings  from  the  first  round  of  surveys  administered  to  teachers  and 
principals  in  the  spring  of  1999.  At  this  time,  elementary  schools  had 
already  administered  the  Washington  Assessment  of  Student  Learning 
(WASL)  for  two  years  (voluntary  in  1996-97  and  mandatory  in  1997-98) 
and  middle  schools  had  administered  the  test  for  one  year  (voluntary 
in  1997-98).  Thus,  the  results  describe  principals'  and  teachers'  early 
responses  to  the  state  education  reform.  The  findings  are  based  on 
principal  and  teacher  self-reports  about  actions  taken  in  their  districts, 
schools,  and  classrooms,  as  well  as  their  opinions  about  the  reform. 
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Washington  Took  a  Gradual  Approach 
to  Standards-Based  Reform 


Education  Reform  Act  passed  (1993) 

Standards  adopted 

-  Reading,  writing,  mathematics,  and  listening  (1995) 

-  Social  studies,  science,  arts,  and  health/fitness  (1996) 
Reading,  writing,  mathematics,  and  listening  tests 
administered 

-  Grade  4:  voluntary  1997;  mandatory  1998 

-  Grade  7:  voluntary  1998;  mandatory  2000 
Accountability  system  scheduled  to  be  adopted  (2000) 

Social  studies,  science,  arts,  and  health/fitness 
assessments  scheduled  to  be  introduced  (2001-08) 


RAND 


In  1993,  the  Washington  state  legislature  passed  the  Student  Learning 
and  Improvement  Act,  which  is  referred  to  as  the  Education  Reform 
Act.  The  legislation  created  a  standards-based  reform  system, 
including  statewide  standards  for  what  students  should  know  and  be 
able  to  do  in  six  subjects,  called  the  Essential  Academic  Learning 
Requirements  (EALRs);  tests  to  evaluate  student  knowledge  and 
progress  toward  standards,  called  the  Washington  Assessment  of 
Student  Learning  (WASL);  and  an  as-yet-to-be-developed  mechanism 
to  hold  schools  accountable  for  student  performance. 

Washington  policymakers  have  adopted  a  gradual  approach  to 
implement  standards-based  reform.  Unlike  many  states — including 
Texas,  Kentucky,  and  North  Carolina — that  implemented  standards- 
based  reforms  rapidly,  Washington  is  introducing  its  reform  over  a 
period  of  a  decade.  For  example,  the  EALRs  for  reading,  writing, 
mathematics,  and  listening  were  developed  first  in  1995.  The  EALRs 
for  science,  social  studies,  health/fitness,  and  the  arts  followed  in  1996. 
The  assessments  were  developed  next,  and  their  full  implementation 
will  take  almost  a  decade.  The  fourth  grade  WASL  in  reading,  writing, 
mathematics,  and  listening  was  offered  for  the  first  time  on  a 
voluntary  basis  in  1996-97,  and  it  became  mandatory  the  following 
school  year.  For  seventh  grade  students,  the  assessments  were 
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voluntary  in  1997-98,  and  will  be  mandatory  in  the  2000-01  school  year. 
The  complete  assessment  system  will  be  fully  implemented  by  2008. 

Additionally,  the  legislature  provided  funds  for  professional 
development,  which  supported  small  discretionary  grants  to  schools 
and  to  16  Regional  Learning  and  Assessment  Centers  around  the  state. 
These  centers  offered  training  on  the  reform.  The  Commission  on 
Student  Learning,  which  was  responsible  for  developing  the  reform 
components,  provided  materials — such  as  Example  Tests  and 
Assessment  Tool  Kits  for  classroom-based  assessment — to  help 
teachers  understand  the  reform. 
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Initial  WASL  Scores  Were  Low, 

But  There  Have  Been  Some  Increases 


Subject 

Fourth  arade 

1997  1998  1999 

Seventh  arade* 

1997  1998  1999 

Mathematics 

21 

31 

37 

20 

24 

Reading 

48 

56 

59 

38 

41 

Writing 

43 

37 

33 

31 

37 

Listening 

62 

71 

71 

80 

87 

*Grade  7  WASL  was  introduced  in  1998  on  a  voluntary  basis. 
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The  early  results  from  the  WASL  showed  that  only  a  minority  of 
students  achieved  the  rigorous  standards  embodied  in  the  state 
reform;  this  is  similar  to  the  results  of  standards-based  tests  in  other 
states.  In  Washington,  fewer  than  one-quarter  of  the  students  met  the 
standards  in  mathematics  in  the  first  year  that  WASL  was 
administered.  Fewer  than  one-half  met  the  standards  in  reading  or 
writing.  Districts  and  schools  felt  strong  pressure  to  do  something  to 
raise  the  scores  after  the  first  round  of  testing.  The  most  recent  WASL 
results  were  more  encouraging,  showing  gains  in  the  percentage  of 
students  meeting  the  standards  in  mathematics,  reading,  and  listening 
in  elementary  and  middle  schools. 
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Multi-State  Study  of 
Standards-Based  Education  Reforms 


RAND  component 

-  The  impact  of  the  reform  on  schools  and  teachers 

-  Statewide  surveys  and  state  data 

CU-Boulder  component 

-  Exemplary  teachers’  responses  to  reform 

-  Case  studies 

Both  build  on  research  conducted  in  Kentucky, 
1994-1998 

-  High-stakes,  performance-based  assessment 

-  Changes  in  practice,  but  questionable  score  gains 


RAND 


RAND  and  CU-Boulder  are  partners  in  this  research  effort.  RAND 
researchers  are  examining  the  impact  of  reform  on  schools  and  classrooms 
through  statewide  surveys  of  principals  and  teachers  and  through  analysis 
of  state  data  (e.g.,  test  results  and  demographic  data).  CU-Boulder 
researchers  are  conducting  case  studies  of  exemplary  elementary  and  middle 
school  writing  and  mathematics  teachers  to  understand  their  response  to 
reform. 

Prior  research  in  Kentucky  indicates  that  standards-based  reforms  that 
include  high-stakes  testing  can  be  powerful  tools  to  change  what  is 
happening  in  schools  and  classrooms.  The  Kentucky  educational  reform 
was  characterized  by  a  high-stakes,  performance-based  testing  system  called 
the  Kentucky  Instructional  Results  Information  System  (KIRIS),  which  was 
linked  to  standards.  The  state  also  provided  extensive  professional 
development  and  support  through  a  network  of  regional  and  local 
organizations.  Kentucky  was  one  of  the  first  states  to  adopt  this  approach, 
and  it  had  been  in  place  for  five  or  six  years  at  the  time  the  RAND  and  CU- 
Boulder  researchers  studied  it. 
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The  research  found  that  Kentucky's  education  reform  influenced 
classroom  practices  in  both  elementary  and  middle  schools.  The  study 
found  evidence  of  increased  professional  development  related  to  the 
tests  and  standards  and  increased  attention  to  the  areas  of  mathematics 
emphasized  by  the  standards  (e.g.,  problem  solving  and  mathematical 
communication).  Researchers  found  no  associations  between 
particular  teaching  practices  and  increased  KIRIS  scores. 

However,  Kentucky  teachers  appeared  to  focus  more  on  the  tests  than 
on  the  standards  the  tests  were  supposed  to  represent.  Teachers 
reallocated  instructional  time  to  subjects  tested  by  KIRIS;  curriculum 
coverage  varied  significantly  from  one  grade  to  the  next  in  parallel 
with  the  subject  matter  emphasis  of  KIRIS.  For  example,  students  in 
fourth  and  seventh  grades  received  more  instruction  in  reading, 
writing,  and  science  (the  subjects  tested  at  grades  4  and  7),  while 
students  in  fifth  and  eighth  grade  received  more  instruction  in 
mathematics,  social  studies,  and  arts/humanities  (the  subjects  tested  at 
grades  5  and  8).  Similar  shifts  in  emphasis  occurred  within  specific 
subject  areas.  For  example,  the  KIRIS  writing  test  focuses  on  short 
written  pieces,  and  teachers  emphasized  that  type  of  writing  at  the 
expense  of  other  types  of  writing. 
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Surveys  of  Washington  Principals  and  Teachers 

•  Representative  sample  of  elementary  and  middle 
schools 

-  70  elementary  schools,  70  middle  schools 

-  All  principals  in  sampled  schools 

-  1 80  fourth  grade  teachers 

-  220  seventh  grade  teachers  of  writing  or 
mathematics 

•  Administered  in  Spring  1999 

•  Response  rate:  77  percent  of  principals,  75  percent 

of  fourth  grade  teachers,  65  percent  of  seventh  grade 
teachers _ _ 

RAND 


The  survey  of  Washington  educators  was  distributed  to  a  stratified 
random  sample  of  70  elementary  and  70  middle  schools  in  Washington 
state  in  spring  1999,  prior  to  the  annual  administration  of  WASL.  The 
schools  in  the  sample  resembled  the  state  in  terms  of  student 
demographics  and  previous  WASL  scores,  so  findings  from  this  study 
should  generalize  to  the  state  as  a  whole. 

Surveys  were  mailed  to  all  principals  in  the  sampled  schools  and  to 
teachers  who  taught  mathematics  and/or  writing  in  the  WASL-tested 
grades  (fourth  grade  and  seventh  grade).  Surveys  were  distributed  to 
180  fourth  grade  teachers  and  220  seventh  grade  teachers.  Along  with 
the  survey,  teachers  received  a  $10  gift  certificate  that  could  be  used  to 
purchase  books  or  materials. 

Survey  response  rates  were  high  for  both  groups.  Seventy-seven 
percent  of  principals  returned  completed  surveys.  Seventy-five 
percent  of  fourth  grade  teachers  and  65  percent  of  seventh  grade 
teachers  also  completed  their  surveys. 
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Those  principals  and  teachers  who  responded  were  representative  of 
principals  and  teachers  in  the  state  as  a  whole.  The  median  number  of 
years  of  service  as  principal  was  seven  for  elementary  school 
principals  who  responded  to  the  survey  and  nine  for  middle  school 
principals.  The  median  tenure  as  principal  at  their  current  school  was 
three  years  for  both  elementary  and  middle  school  principals.  The 
median  elementary  teacher  who  responded  to  the  survey  had  17  years 
of  teaching  experience,  with  seven  of  these  at  his/her  current  school. 
The  median  middle  school  teacher  had  nine  years  of  experience,  four 
at  his/her  current  school.  About  one-half  of  the  teacher  respondents 
had  master's  degrees. 
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Survey  Topics  | 

Principals 

Teachers  p 

•  Background 

•  Background  k 

•  Familiarity  with  reform 

•  Familiarity  with  reform  t. 

•  Opinions  about  reform 

•  Opinions  about  reform  f 

•  Participation  in 

•  Participation  in  I 

professional  development 

professional  development  p 

•  Implementation  in 

•  Changes  in  classroom  p 

districts  and  schools 

practices  in  writing  and  k 

mathematics  | 

RAND  I 

The  principal  and  teacher  surveys  covered  a  range  of  issues  related  to  the 
Washington  education  reform.  Both  principals  and  teachers  were  asked 
about  their  own  educational  background.  In  addition,  they  responded  to 
questions  about  the  professional  development  they  received,  their 
familiarity  with  the  education  reform,  and  their  opinions  about  it.  Both 
groups  were  also  asked  about  educational  practices  in  their  schools  and 
about  changes  that  had  occurred  since  1997-98. 

Specifically,  principals  responded  to  questions  about  policy  changes  at 
the  district  and  school  levels,  while  teachers  were  asked  about  specific 
changes  in  their  classroom  practices.  Teacher  questions  included  their 
allocation  of  time  to  different  subjects,  the  topics  they  emphasized  in 
mathematics  and  writing,  and  their  teaching  strategies.  Teachers  also 
rated  the  influence  of  different  elements  of  the  state  reform  on  their 
classroom  practices.  All  the  findings  relating  to  school  and  classroom 
practices  and  principal  and  teacher  opinions  are  based  on  self-reported 
responses  to  these  surveys. 
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Major  Findings 


•  Principals  and  teachers  understood  and  endorsed  the 
reform 

•  District  and  school  policies  changed  to  support  the 
reform 

•  Local  standards  and  curriculum  changed  to  reflect 
EALRs 

•  Classroom  practices  changed  to  emphasize  WASL 

•  Certain  practices  were  associated  with  higher  WASL 
scores 

•  Students  engage  in  extensive  test  preparation 
activities 


The  surveys  revealed  a  picture  of  schools  in  transition,  with  many  changes 
occurring  widely,  but  not  uniformly,  across  the  state.  Briefly,  the  study 
found  that  principals  and  teachers  in  Washington  spent  a  great  deal  of  time 
learning  about  the  reform,  and  they  believed  they  understood  its  key 
components  well.  For  example,  they  believed  the  standards  are  appropriate 
and  attainable. 

A  variety  of  changes  occurred  as  a  result  of  the  reform.  At  the  district  level, 
districts  took  steps  to  align  curriculum  and  assessment  with  the  EALRs. 
Schools  and  teachers  also  worked  to  make  their  programs  consistent  with 
the  direction  set  by  the  state.  Teachers  changed  classroom  curriculum  and 
instruction  in  response  to  the  reform.  In  general,  classroom  changes 
appeared  to  be  consistent  with  the  EALRs,  but  local  educators  appeared  to 
be  responding  more  strongly  to  the  highly  visible  WASL  scores  than  to  the 
EALRs.  For  example,  teachers  shifted  their  emphasis  in  the  classroom  to  the 
WASL-tested  subjects. 

Researchers  found  that  a  school's  WASL  scores  were  positively  related  to 
teachers'  knowledge  of  WASL  and  the  alignment  of  the  school's  curriculum 
with  the  EALRs.  They  also  found  that  teachers  engaged  in  test  preparation 
activities  that  they  believed  were  largely  responsible  for  WASL  score  gains. 
These  two  points  illustrate  some  of  the  questions  the  study  raises  about  the 
meaning  of  WASL  scores  and  score  gains. 

The  findings  will  be  elaborated  in  subsequent  slides. 
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Principals  and  Teachers  Understood 
and  Endorsed  the  Reform 


Received  50-80  hours  of  professional  development 
(in  two  years),  about  one-half  related  to  reform 

More  than  one-half  of  principals  and  teachers 

-  Served  on  district  and  school  committees  implementing  the 
reform 

-  Believed  they  understood  EALRs  and  WASL 

-  Believed  the  reform  goals  and  EALRs  were  appropriate 

Principals  were  more  optimistic  about  the  reform 
than  teachers 


RAND 


Jmsm 


Principals  and  teachers  spent  considerable  time  learning  about 
the  reform  during  the  past  two  years.  The  median  number  of 
hours  principals  spent  in  in-service  training  or  formal 
professional  development  during  the  past  two  years  was  80 
hours,  and  one-half  or  more  of  this  time  was  related  to  the 
reform.  Similarly,  the  median  number  of  hours  teachers 
participated  in  in-service  training  or  formal  professional 
development  was  50  hours,  and  one-half  was  related  to  the 
reform.  This  translates  into  three  to  five  days  of  professional 
development  per  year. 

Many  principals  and  teachers  also  served  on  district  or  school 
committees  working  on  standards,  curriculum  alignment  with 
standards,  or  classroom-based  assessment  materials,  which 
contributed  to  their  understanding  of  the  reform.  As  a  result, 
more  than  three-quarters  of  principals  and  one-half  of  teachers 
indicated  that  they  understood  the  reform  somewhat  well  or  very 
well.  Furthermore,  both  groups  endorsed  key  elements  of  the 
reform.  Three-quarters  of  principals  and  one-half  of  teachers 
agreed  that  the  EALRs  were  appropriate  for  benchmark  grade 
levels  and  the  goals  of  the  reform  were  attainable. 
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However,  principals  were  somewhat  more  enthusiastic  about  the 
reform  than  teachers.  For  example,  87  percent  of  elementary  principals 
but  only  61  percent  of  fourth  grade  teachers  believed  EALRs  were 
appropriate.  Similarly,  principals  were  much  more  optimistic  than 
teachers  about  the  status  of  implementation.  More  principals  than 
teachers  agreed  that  curriculum  in  their  schools  was  aligned  with  the 
EALRs  and  that  each  of  the  components  of  reform  promoted  better 
instruction  and  increased  student  learning.  (The  differences  ranged 
from  10  to  20  percentage  points.) 

There  are  a  number  of  possible  explanations  for  these  differences  of 
opinion.  They  may  reflect  the  fact  that  the  reform  is  being 
implemented  in  a  top-down  manner,  and  principals  are  coming  to 
understand  and  endorse  it  first.  They  may  reflect  the  reluctance  of 
teachers  to  change  classroom  practices.  Or  they  may  reflect  teachers' 
clearer  understanding  of  the  demands  of  the  reform  at  the  classroom 
level. 
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District  and  School  Policies  Changed 

_ to  Support  the  Reform _ 

•  Two-thirds  of  districts  changed  standards, 
and  about  60  percent  changed  local  testing 
programs 

•  Most  schools  actively  conveyed  information 
about  reform  to  teachers 

•  More  than  one-half  of  schools  changed 
schedules  to  increase  time  for  reading, 
writing,  and  mathematics 

•  Some  schools  changed  report  card  format, 
student  promotion  policies 

RAND 


Most  districts  and  schools  made  changes  to  support  the  state  reform. 
Many  districts  changed  their  standards  to  coordinate  local  expectations 
of  what  students  should  know  and  be  able  to  do  with  the  state's 
expectations,  as  articulated  in  the  EALRs.  The  extent  to  which  local 
and  state  standards  are  coordinated,  communicating  the  same  message 
about  reform  goals,  expectations  of  students,  and  student  achievement, 
is  often  referred  to  as  "alignment."  Alignment  of  local  standards  and 
school  curriculum  with  state  reforms  is  a  common  early  step  in  reform. 

Almost  all  districts  took  actions  to  adopt  or  revise  their  local 
curriculum  standards  after  the  EALRs  were  developed.  More  than 
one-half  of  districts  also  changed  local  testing  programs— adding  new 
tests  or  changing  the  content  or  grades  tested  to  align  with  the  state 
reform.  Few  districts  eliminated  any  local  testing.  As  a  result,  the  total 
amount  of  testing  rose  in  both  elementary  and  middle  schools. 

Schools  responded  to  the  WASL  in  a  number  of  ways.  For  example,  all 
elementary  schools  and  almost  all  middle  schools  organized  meetings 
of  teachers  and  staff  to  share  information  about  WASL.  Almost  all 
schools  also  offered  professional  development  opportunities  that 
focused  on  WASL-related  issues. 
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Many  schools  also  made  more-fundamental  changes  to  their 
organization  or  policies.  About  one-half  of  the  schools  changed 
schedules  to  increase  or  focus  time  on  tested  subjects.  Some  schools 
also  changed  their  report  card  formats  or  student  promotion  policies. 
Some  schools  offered  explicit  incentives  for  students  to  do  well  on 
WASL,  including  parties  and  field  trips. 
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District  Standards  Changed  to  Reflect  EALRs,  1 

Particularly  in  Tested  Subjects  § 

1 

Elementary 

Middle  ;* 

Subjects 

School 

School  t 

Reading 

94 

96  1 

Writing 

94 

98 

Mathematics 

93 

90  1 

Communication/Listening 

83 

83  r 

Social  Studies 

65 

61  1 

Science 

67 

76  1 

Arts 

56 

61  1 

Health  and  Fitness 

56 

56  1 

Percent  of  principals  indicating  district  standards  are  somewhat  well  If 

or  very  well  aligned  with  EALRs 

1 

The  surveys  asked  principal  "how  well"  their  district  standards  or 
school  curriculum  aligned  with  EALRs,  and  most  principals  believed 
that  their  district's  standards  were  aligned  with  the  EALRs  and  with 
the  WASL  tests.  As  the  first  three  rows  of  the  table  illustrate,  more 
than  90  percent  of  principals  in  elementary  schools  and  middle  schools 
reported  their  districts  standards  to  be  somewhat  or  very  well  aligned 
with  the  EALRs  in  reading,  writing,  and  mathematics.  Over  80  percent 
of  principals  thought  their  district's  assessments  were  aligned  with 
WASL  in  communication  and  listening,  as  well.  In  contrast,  a  lower 
percentage  of  principals  felt  their  district's  standards  were  aligned 
with  the  EALRs  in  subjects  not  currently  tested  by  WASL. 


18 


School  Curriculum  Was  Aligned  with  EALRs, 
Particularly  in  Tested  Subjects 


Subject 

Elementary  School 
Principals  Teachers 

Middle  School 
Principals  Teachers 

Reading 

100 

82 

96 

96 

Writing 

98 

89 

100 

93 

Mathematics 

94 

81 

90 

81 

Communication/Listening 

77 

62 

85 

90 

Social  Studies 

64 

52 

64 

75 

Science 

68 

52 

79 

- 

Arts 

51 

35 

60 

-- 

Health  and  Fitness 

57 

37 

65 

~ 

■  - -  i - 1 - 

or  very  well  aligned  with  EALRs 


Over  80  percent  of  principals  and  teachers  reported  that  their  school 
curriculum  was  aligned  with  the  EALRs  in  the  WASL-tested  subjects  (with 
the  exception  of  communication/listening).  This  table  shows  both  groups' 
perceptions  of  alignment  across  the  different  subjects. 

Principals  were  more  certain  than  teachers  that  curriculum  aligned  with 
EALRs  in  the  WASL-tested  subjects.  The  percentage  of  both  principals  and 
teachers  reporting  alignment  was  lower  in  subjects  not  tested  by  WASL — 
social  studies,  science,  arts,  and  health/fitness— than  in  subjects  tested  by 
WASL. 

Teachers  also  reported  that  their  curriculum  materials  were  aligned  with  the 
EALRs.  In  writing,  for  example,  slightly  more  than  one-half  of  the  fourth 
grade  teachers  and  about  three-quarters  of  the  seventh  grade  writing  teachers 
indicated  that  they  used  a  writing  or  language  arts  textbook.  Approximately 
29  percent  of  these  teachers  thought  their  textbook  was  very  well  aligned 
with  the  EALRs  in  their  subject. 
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Classroom  Practices  Changed 
to  Emphasize  WASL 


•  Teachers  reallocated  instructional  time  to 
tested  subjects 

•  Teaching  practices  and  instruction  became 
somewhat  more  consistent  with  EALRs 

•  WASL  had  the  greatest  influence  on  practice 

•  Both  EALRs  and  WASL  were  more  influential 
than  voluntary  classroom-based 
assessments 


RAND 


Teachers  have  changed  their  curriculum  and  instruction  since  the 
introduction  of  Washington  education  reform.  Changes  occurred  in  the 
allocation  of  classroom  time,  curriculum  content,  and  teaching  methods. 
For  example,  teachers  and  schools  reallocated  instructional  time  to 
emphasize  subjects  tested  by  WASL  at  the  expense  of  subjects  not  tested 
by  WASL.  Similarly,  teachers  and  principals  reported  that  curriculum 
and  instruction  are  more  aligned  in  tested  subjects  than  in  non-tested 
subjects. 

In  general,  changes  in  curriculum  and  instruction  appeared  to  be 
consistent  with  the  EALRs.  For  example,  teachers  increased  their 
attention  to  different  genres  in  writing;  something  that  was  not  common 
prior  to  the  introduction  of  the  EALRs.  However,  local  educators 
appeared  to  respond  most  to  the  highly  visible  WASL  tests.  For  example, 
teachers  increased  the  use  of  rubrics  in  writing  and  extended-response 
items  in  mathematics,  both  of  which  are  prominent  features  of  the  tests. 

Finally,  we  found  that  not  all  aspects  of  the  reform  were  equally  salient 
to  teachers.  Classroom-based  assessment,  in  particular,  was  not  as 
widely  understood  or  as  influential  as  the  EALRs  or  WASL. 
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Teachers  Reallocated  Instructional  Time 
to  Emphasize  WASL-tested  Subjects 


80  60  40  20  0  20  40  60 

Percent  of  4th  grade  teachers  who  changed  instructional  time 


Health  &  Fitness 
Arts 

Science 

Social  Studies 

Communication/ 

Listening 

Mathematics 

Writing 

Reading 

Total 


80 


The  most  dramatic  change  in  classroom  practices  was  the  reallocation 
of  instructional  time.  Fourth  grade  teachers,  who  teach  all  subjects, 
reported  marked  decreases  in  time  spent  on  subjects  not  included  in 
the  WASL  and  marked  increases  in  subjects  tested  by  the  WASL. 

Also,  about  20  percent  of  teachers  indicated  that  overall  instructional 
time  had  increased. 

These  reports  are  similar  to  the  findings  in  Kentucky,  where  teachers 
spent  more  time  on  tested  subjects  at  the  expense  of  non-tested 
subjects.  As  noted  previously,  surveys  of  Kentucky  teachers  revealed 
that  curriculum  coverage  varied  significantly  from  one  grade  to  the 
next  in  parallel  with  the  subject  matter  emphasis  of  KIRIS. 

The  shift  in  instructional  time  toward  tested  subjects  may  decline  as 
Washington  introduces  WASL  tests  in  science,  social  studies,  arts,  and 
health/fitness  over  the  next  seven  years.  However,  the  state  plans  to 
introduce  these  tests  in  grades  other  than  the  benchmark  grades  at  the 
elementary  and  middle  school  levels.  Splitting  the  WASL  tests 
between  two  grades  distributes  the  testing  burden  among  grade  levels. 
However,  it  also  creates  an  incentive  for  teachers  to  focus  their 
instruction  on  the  tested  subjects  at  the  expense  of  non-tested  subjects; 
thus  the  additional  grades  tested  may  also  suffer  from  reallocations  of 
instructional  time  to  emphasize  the  tested  subjects. 
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Writing  Curriculum  Emphasized  the  “Writing 
Process,”  but  Genre  Increased  Most 


Selected  Aspects  of  Writing 

Covered  Aspect 
Dailv  or  Weeklv 
Grade  4  Grade  7 

Increased  Coverage 
Durina  Past  2  Years 
Grade  4  Grade  7 

Application  of  writing  conventions 

86 

83 

37 

46 

Writing  process:  draft 

73 

65 

34 

35 

Writing  process:  edit 

68 

57 

36 

32 

Genre:  style  appropriate  to  audience 

42 

32 

51 

60 

and  purpose 

Genre:  write  for  different  audiences 

28 

22 

43 

53 

Percent  of  teachers 


RAND 


This  slide  illustrates  two  things:  the  frequency  of  coverage  of 
different  elements  of  the  writing  curriculum,  and  the  change  in 
frequency  during  the  past  two  years.  Looking  first  at  coverage,  we 
found  that  the  content  of  the  writing  curriculum  was  broadly  reflective 
of  the  EALRs  in  both  the  fourth  and  seventh  grades.  Eleven  of  the  14 
writing  behaviors  specified  in  the  EALRs  were  covered  weekly  or 
more  often  by  over  40  percent  of  the  teachers  at  both  grade  levels. 
However,  as  shown  in  this  slide,  teachers  concentrated  classroom  time 
more  on  writing  conventions  and  on  the  writing  process  than  on  the 
other  elements  of  the  EALRs.  (Note:  the  slide  contains  only  a  selected 
subset  of  the  items  on  the  survey.) 

In  response  to  a  general  question  about  change,  most  writing  teachers 
in  the  fourth  and  seventh  grades  reported  that  they  changed  the 
overall  content  of  their  writing  lessons  and  their  teaching  methods 
during  the  last  two  years.  At  the  fourth  grade  level,  42  percent  of 
teachers  reported  a  great  deal  of  change,  and  81  percent  of  teachers 
reported  making  at  least  a  moderate  amount  of  change.  Fewer  seventh 
grade  writing  teachers  made  changes:  29  percent  reported  a  great  deal 
of  change  and  55  percent  reported  at  least  a  moderate  amount  of 
change. 


22 


More  specifically,  the  greatest  percentage  of  teachers  increased 
coverage  of  the  less-traditional  topics.  Roughly  one-half  of  the 
teachers  in  both  grades  reported  increasing  their  coverage  of  different 
audiences,  purposes,  and  forms  of  writing,  as  well  as  the  application 
of  styles  appropriate  to  different  audiences  and  purposes. 

Thus,  the  writing  content  most  frequently  taught  remained  unchanged 
over  time  while  teachers  increased  the  frequency  with  which  they 
taught  the  other  content  areas.  Content  most  frequently  taught  might 
be  considered  more  "traditional"  while  the  other  content  appears  to  be 
more  "reform-oriented." 
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Writing  Instruction  Emphasized  Mechanics, 
but  Rubric-based  Instruction  Increased  Most 


Used  Teaching  Strategy 

Increased  Use 

Dailv  or  Weekly 

Durina  Past  2  Years 

Writinq  Instructional  Strategy 

Grade  4 

Grade  7 

Grade  4 

Grade  7 

Read  orally  to  students 

97 

76 

13 

30 

Explain  correct  usage  of  grammar, 

90 

86 

20 

46 

spelling,  punctuation,  and  syntax 
Teach  Six-Trait  or  other  rubric-based 

64 

41 

56 

61 

approach  to  writing 

Provide  time  for  unstructured  (“free11) 

53 

40 

14 

25 

writing 

Comment  on  student  writing  in  terms 

30 

31 

62 

69 

of  WASL  rubrics 

Percent  of  teachers 


RAND 


This  slide  illustrates  two  things:  the  frequency  that  teachers  used 
different  writing  instructional  strategies,  and  the  change  in  frequency 
during  the  past  two  years.  On  the  survey,  teachers  reported  the 
frequency  with  which  they  used  15  different  instructional  strategies 
ranging  from  fairly  traditional  techniques,  for  example,  "read  orally  to 
students,"  to  more  innovative  approaches,  for  example,  "write  with 
students  on  the  same  assignment."  Responses  to  a  subset  of  five  of 
these  strategies  are  summarized  here.  Most  teachers  in  both  grades 
read  to  students  at  least  once  a  week  and  taught  language  mechanics 
(grammar,  spelling,  punctuation  and  syntax)  as  frequently. 

However,  the  area  of  greatest  change  in  writing  instruction  was 
teachers'  use  of  rubrics.  More  than  one-half  of  teachers  increased  the 
frequency  with  which  they  comment  on  student  writing  in  terms  of 
WASL  rubrics  and  their  use  of  Six-Trait  or  other  rubric-based 
approaches  to  writing. 
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Mathematics  Curriculum  Emphasized  Number  1 
Sense,  but  Probability  and  Statistics  Increased  Most  1 

Mathematics  Content  Area 
(from  EALRs) 

Covered  Aspect  Increased  Coverage  g 

Dailv  or  Weeklv  Durina  Past  2  Years  18 

Grade  4  Grade  7  Grade  4  Grade  7  H 

1.1  Number  Sense 

76 

79  29 

29  i 

1.2  Algebraic  Sense 

30 

53  31 

37 

1.3  Measurement 

29 

21  28 

18 

1.4  Geometric  Sense 

27 

20  32 

35 

1.5  Probability  and  Statistics 

17 

16  40 

52 

Percent  of  teachers 

RAND  1 

Teachers'  descriptions  of  their  mathematics  curriculum  and  teaching 
strategies  revealed  a  picture  similar  to  writing,  although  not  quite  as 
dramatic.  This  slide  shows  curriculum  coverage;  the  next  slide  shows 
teaching  methods. 

Of  the  five  major  content  areas  of  mathematics,  number  sense  was 
covered  at  least  weekly  by  the  greatest  percentage  of  teachers.  In 
contrast,  probability  and  statistics  was  covered  weekly  by  the  lowest 
percentage  of  teachers. 

Mathematics  content  has  not  changed  as  much  as  writing  content  in 
the  past  two  years.  With  one  exception,  only  about  one-third  of  the 
teachers  reported  increasing  coverage  of  any  of  the  five  topics.  The 
exception  was  in  the  area  of  probability  and  statistics;  about  one-half 
of  the  mathematics  teachers  increased  their  coverage  of  this  topic 
during  the  past  two  years.  In  addition,  more  fourth  grade  than  seventh 
grade  teachers  increased  their  coverage  of  measurement,  while  more 
seventh  grade  teachers  than  fourth  grade  teachers  increased  their 
coverage  of  probability  and  statistics. 
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Mathematics  instruction  Included  Many  Strategies, 
but  Open-Response  Questions  Increased  Most 


Mathematics  Teachinq  Strategy 

Used  Teaching  Strategy 
Dailv  or  Weeklv 
Grade  4  Grade  7 

Increased  Use 
Durina  Past  2  Years 
Grade  4  Grade  7 

Explain  correct  solutions 

91 

90 

30 

12 

Demonstrate  new  skill 

81 

81 

26 

22 

Assess  students'  mathematics  skills 

73 

76 

29 

19 

Give  examples  of  real-life  applications 

76 

73 

47 

44 

Ask  open-response  questions  with 

83 

72 

65 

58 

many  right  answers 

Demonstrate  mathematical  ideas  using 

61 

33 

41 

18 

constructions,  manipulatives,  etc. 

Conduct  speed  drills 

54 

12 

14 

6 

Most  mathematics  teachers  used  a  range  of  instructional  strategies  oh  a 
regular  basis.  More  than  70  percent  of  mathematics  teachers  in  grades 
four  and  seven  explained  new  concepts,  demonstrated  new  skills, 
assessed  student  skills,  gave  examples  of  real-life  applications,  and 
asked  open-response  questions  at  least  weekly. 

Fewer  teachers  regularly  conducted  speed  drills  or  used 
manipulatives,  particularly  in  middle  school.  Only  12  percent  of 
seventh  grade  mathematics  teachers  used  speed  drills  regularly.  In 
fact,  some  teachers,  including  29  percent  of  seventh  grade  teachers, 
reported  that  they  never  used  speed  drills.  Similarly,  only  one-third  of 
seventh  grade  mathematics  teachers  had  students  demonstrate  their 
mathematical  skills  using  manipulatives  at  least  weekly. 

The  influence  of  the  education  reform  may  be  seen  in  the  changes 
teachers  made  in  their  mathematics  teaching  strategies.  More  than 
one-half  of  teachers  asked  more  open-response  questions  with  many 
right  answers.  Also,  more  than  40  percent  of  teachers  increased  their 
use  of  examples  of  real-life  applications  of  mathematics.  A  related 
finding  (not  reported  on  this  slide)  was  that  22  percent  of  mathematics 
teachers  decreased  their  use  of  speed  drills. 
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Certain  Practices  Were  Associated  with 
Higher  WASL  Scores _ 


•  WASL  scores  were  higher  in  schools  where: 

-Curricula  aligned  with  EALRs  (reading, 
mathematics) 

-  Teachers  understood  EALRs  and  WASL 
(reading) 

•  Controlling  for  school  size  and  student 
demographics 


The  researchers  used  a  statistical  technique  called  regression  analysis 
to  investigate  the  relationships  between  WASL  scores  and  survey 
responses,  while  also  accounting  for  differences  in  school  size  and 
student  demographics.  Researchers  examined  selected  items  from  the 
principal  and  teacher  surveys  related  to  curriculum  alignment,  teacher 
understanding,  school  policies  and  actions  in  support  of  the  reform, 
and  test  preparation.  They  compared  responses  to  these  items  to 
schools'  WASL  scores. 

There  was  a  statistically  significant  positive  relationship  between 
WASL  scores  and  curriculum  alignment.  Schools  where  principals 
and  teachers  judged  curriculum  to  be  more  aligned  with  the  EALRs 
had  higher  WASL  scores  in  reading,  mathematics,  and  listening  than 
schools  whose  curriculum  was  judged  to  be  less  aligned  with  the 
EALRs.  Schools  where  teachers  reported  more  understanding  of  the 
EALRs  and  WASL  had  higher  WASL  scores  in  mathematics,  (and 
marginally  higher  scores  in  reading)  than  schools  where  teachers 
reported  less  understanding  of  the  EALRs. 

These  results  are  noteworthy  because  it  is  unusual  to  find  such 
relationships  when  using  school  average  scores  and  school  average 
survey  responses.  They  suggest  that  alignment  is  more  than  just  a 
"catch-phrase"  but  an  important  element  in  responding  to  the 
Washington  education  reform. 
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Teachers’  Opinions  About  WASL  Raise  Concerns 


Teachers  agreed  that: 

•  Test  preparation  is  responsible  for  most  WASL  score 
gains 

•  The  WASL  tests  are  not  of  appropriate  difficulty  for 
the  tested  grade  levels  (except  seventh  grade 
writing) 

•  Differences  in  students  from  one  year  to  the  next 
make  it  difficult  to  prepare  students  for  WASL 

•  It  is  not  easy  to  raise  WASL  scores  by  focusing  on  a 
few  specific  skills 

_  _  —  rand 


On  the  survey,  teachers  were  asked  to  agree  or  disagree  with  a  number 
of  statements  about  the  education  reform;  four  of  these  statements 
were  directly  related  to  the  WASL  test.  Teachers'  responses  to  these 
items  raise  some  concerns  about  the  meaningfulness  of  WASL  scores. 
About  three-quarters  of  fourth  grade  teachers  and  seventh  grade 
mathematics  teachers  disagreed  with  the  statement  that  the  "WASL 
tests  are  of  the  appropriate  difficulty  for  the  tested  grades." 
Furthermore,  most  teachers  agreed  that  changes  in  WASL  scores  could 
be  attributed  to  factors  that  were  unrelated  to  changes  in  students' 
mastery  of  the  standards.  Specifically,  teachers  agreed  that  better  test 
preparation  is  responsible  for  most  of  the  changes  in  WASL  scores." 
Teachers  also  reported  that  they  increased  their  use  of  sample  test 
items  and  WASL-like  questions  in  class. 

On  the  other  hand,  few  teachers  thought  they  could  easily  influence 
the  WASL  scores,  that  is,  "game"  the  test.  Fewer  than  one-half  of 
teachers  agreed  with  the  statement  "it  is  easy  to  raise  (test)  scores  by 
focusing  narrowly  on  a  few  specific  skills"  featured  on  the  test.  More 
than  70  percent  of  teachers  reported  that  "differences  in  student 
characteristics  from  year  to  year  make  it  difficult  to  prepare  students 
for  WASL." 
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Increased  test  preparation  activities  raise  complex  questions  about  the 
meaning  of  the  WASL  results.  Some  types  of  test  familiarization  are 
quite  appropriate  and  help  students  produce  scores  that  are  indicative 
of  their  broader  abilities.  Other  types  of  focused  preparation  are  quite 
inappropriate.  By  focusing  narrowly  on  the  particular  content  or 
format  of  the  test,  teachers  may  increase  students'  WASL  scores 
without  increasing  their  broad  knowledge  of  the  EALRs.  This  issue 
deserves  continued  scrutiny  in  the  future. 
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Study  Reveals  that  Implementation  is 
Occurring,  but  Raises  Some  Concerns 

•  Educators  generally  understood  and  supported  the 
reform 

•  Changes  in  policy,  school  organization,  curriculum, 
and  teaching  were  consistent  with  the  reforms 

•  WASL  strongly  influenced  many  of  the  changes 

•  Schools  where  educators  reported  more  curriculum 
alignment  had  higher  WASL  scores 

•  Teachers  reallocated  instructional  time  from  non- 
tested  to  tested  subjects 

•  Extensive  test  preparation  activities  need  further 
examination 

RAND 


In  summary,  this  study  shows  that  educators  responded  to  the  Washington 
education  reform  with  changes  at  the  district,  school,  and  classroom  levels. 
Principals  and  teachers  devoted  considerable  effort  to  learning  about  the 
reform  and  adapting  school  practices  to  support  it.  Principals  and  teachers 
reported  that  they  understood  and  supported  the  reform.  Districts  changed 
standards,  curriculum,  and  assessments  to  be  consistent  with  the  state 
reform.  Similarly,  teachers  made  modest  changes  in  curriculum  content  and 
instructional  strategies  in  writing  and  mathematics  to  promote  the  goals  of 
the  reform.  WASL  appeared  to  be  the  most  salient  influence  on  teachers  and 
schools.  Teachers  indicated  that  many  of  their  changes  in  the  writing  and 
mathematics  curriculum  and  instruction  mirrored  the  content  or  format  of 
the  WASL  tests.  Other  things  being  equal,  schools  with  greater  curriculum 
alignment  also  had  higher  WASL  scores  in  reading  and  mathematics. 

Teachers  made  some  changes  that  may  be  less  desirable.  In  some  cases, 
teachers  emphasized  tested  material  and  format  over  equally  important 
content  not  covered  by  state  testing.  For  example,  teachers  increased  the 
amount  of  instructional  time  spent  on  tested  subjects  at  the  expense  of  non- 
tested  subjects.  Similarly,  educators  reported  that  the  alignment  of  local 
standards  and  curriculum  with  EALRs  was  higher  in  the  tested  subjects. 


30 


Finally,  teachers  spent  considerable  time  specifically  preparing 
students  for  the  WASL.  Such  actions  deserve  continuing  examination 
to  ensure  that  they  do  not  detract  from  the  goals  of  the  reform. 

A  follow-up  survey  of  Washington  educators  was  administered  in 
spring  2000.  The  survey  covers  the  same  topics  in  order  to  monitor 
changes  in  the  implementation  of  the  reform.  The  results  of  the  2000 
survey  will  be  available  in  spring  2001. 
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